Adaptive sensing based on depth

ABSTRACT

A microscope for adaptive sensing may comprise an illumination assembly, an image capture device configured to collect light from a sample illuminated by the assembly, and a processor. The processor may be configured to execute instructions which cause the microscope to capture, using the image capture device, an initial image set of the sample, identify, in response to the initial image set, an attribute of the sample, determine, in response to identifying the attribute, a three-dimensional (3D) process for sensing the sample, and generate, using the determined 3D process, an output image set comprising more than one focal plane. Various other methods, systems, and computer-readable media are also disclosed.

RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/IL2018/051117, filed Oct. 18, 2018, published as WO 2019/077610 onApr. 25, 2019, which claims the benefit under 35 U.S.C. § 119(e) of U.S.Provisional Application No. 62/574,289, filed Oct. 19, 2017, entitled“ADAPTIVE SENSING BASED ON DEPTH”, the disclosures of which areincorporated, in their entirety, by this reference.

BACKGROUND

The present disclosure relates generally to digital microscopy and/orcomputational microscopy and, more specifically, to systems and methodsfor adaptive sensing of a sample.

Today's commercial microscopes may rely on expensive and delicateoptical lenses and typically rely on additional hardware to share andprocess acquired images. Moreover, for scanning optical microscopy,additional expensive equipment such as accurate mechanics and scientificcameras can be utilized. A new generation of microscope technology,known as computational microscopy, has begun to emerge, and makes use ofadvanced image-processing algorithms (usually with hardwaremodifications) to overcome limitations of conventional microscopes. Forexample, microscopic objectives often rely on a high optical resolution,therefore a high numerical aperture which may result in a very shallowdepth of focus. The depth of focus may typically be in the order ofmicrons or less for microscopic applications. However, some samples maynot be thin enough to fit inside the depth of focus of the lens. Thismay result in one focused slice of the sample whereas the other slicesare out of focus slices, which may appear blurred and add noise to theimage. Moreover, more than one plane of focus may be present in the samefield of view if the sample is not flat enough for the lens.

A conventional solution to this issue includes focus stacking (alsoreferred to as Z-stacking). In focus stacking, the sample may bemeasured in a stack of different focal planes for which different slicesof the sample are focused. For example, focus stacking may be based ondefining a range for focus scanning and a region of interest (ROI) ofthe sample, and imaging the entire ROI at the defined range. The sampleis scanned in several focal planes by changing the distance between theoptics and the sample. However, focus stacking may disadvantageously betime consuming, as the measuring or acquisition time is multiplied bythe number of focal planes inside the focus stack. Focus stacking mayrely on increased data storage and with thicker samples, the resolutionand quality of the image may deteriorate. For automatic microscopes,such as slide scanners, acquisition time may take longer than would beideal. Also, for computational microscopy, the sampling time andcomputational time to generate images can be less than ideal.

SUMMARY

As will be described in greater detail below, the instant disclosuredescribes various systems and methods for adaptive sensing of a sampleby detecting an attribute of the sample, and determining an acquisitionprocess in response to the attribute. This adaptive approach to theimage acquisition and computational process can decrease the acquisitiontime, storage requirements, costs and processing times. In addition,larger focus spans may be enabled where they are needed, thus improvingthe quality of the data and reducing a risk of human error in choosingthe area and in analyzing the data.

The presently disclosed systems and methods relate to the fields ofcomputational microscopy and digital microscopy. Some disclosedembodiments are directed to systems and methods for focusing amicroscope using images acquired under a plurality of illuminationconditions. The disclosed embodiments may also comprise systems andmethods for acquiring images under a plurality of illuminationconditions to generate a high-resolution image of a sample. Althoughreference is made to computational microscopy, the methods and apparatusdisclosed herein will find application in many fields, such as 3Dsampling with conventional microscopes, slide scanners and confocalmicroscopy.

In some embodiments, a microscope may comprise an illumination assembly,an image capture device configured to collect light from a sampleilluminated by the illumination assembly, and a processor configured toexecute instructions. The instructions may cause the microscope tocapture, using the image capture device, an initial image set of thesample, identify, in response to the initial image set, an attribute ofthe sample, determine, in response to identifying the attribute, athree-dimensional (3D) process, and generate, using the determined 3Dprocess, an output image set comprising more than one focal plane.

In some embodiments, the 3D process may comprise a process for sensingthe sample. In some embodiments, the 3D process may comprise areconstruction process for reconstructing the sample in response to theinitial image set. In some embodiments, the 3D process may not rely onadditional images beyond the initial image set.

In some embodiments, the 3D process may comprise capturing one or moresubsequent images of the sample using one or more illuminationconditions of the illumination assembly and one or more image capturesettings for the image capture device. In some embodiments a number ofillumination conditions for the 3D process is greater than a number ofillumination conditions for capturing the initial image set. The 3Dprocess may comprise determining a plurality of focal planes forcapturing the one or more subsequent images based at least on theattribute, and the one or more illumination conditions and the one ormore image capture settings correspond to the plurality of focal planes.The one or more subsequent images may be taken at one or more locationsof the sample determined based at least on the attribute.

In some embodiments, the 3D process may comprise performing a 3Dreconstruction of the sample based at least on a subset of imagescaptured by the image capture device. The 3D process may compriseperforming a 2.5D reconstruction of the sample based at least on asubset of images captured by the image capture device in order togenerate 3D data from the sample.

In some embodiments, the 3D process comprises performing focus stackingfor the sample based at least on a subset of images captured by theimage capture device. The 3D process may comprise capturing a pluralityof images at a respective plurality of focal planes.

In some embodiments, identifying the attribute may comprise estimatingthe attribute corresponding to one or more locations of the sample. The3D process may be performed within a threshold time from capturing theinitial image set. The threshold time may be one of 5 microseconds, 10microseconds, 1 second, 5 seconds, 10 seconds, 1 minute, or 5 minutes.

In some embodiments, the attribute may comprise a thickness of thesample at one or more locations of the sample. The attribute maycomprise a depth of the sample at one or more locations of the sample.

In some embodiments, the 3D process may comprise capturing images at aplurality of distances between the image capture device and the sample,and a number of focal planes for the 3D process is greater than a numberof distances in the plurality of distances.

In some embodiments, capturing the initial image set may comprisecapturing the initial image set of the sample using a plurality ofillumination conditions for illuminating the sample, and the pluralityof illumination conditions comprise at least one of an illuminationangle, an illumination wavelength, or an illumination pattern. The 3Dprocess may comprise capturing one or more subsequent images of thesample using a plurality of illumination conditions for illuminating thesample, and the plurality of illumination conditions may comprise atleast one of an illumination angle, an illumination wavelength, or anillumination pattern. The 3D process may comprise a plurality of focuslevels for adjusting the image capture device.

In some embodiments, the attribute may comprise at least one of athickness, a density, a depth, a color, a stain structure of the sample,a distance from a lens of the image capture device, a plurality ofdistances between the lens of the image capture device and the sample, aplurality of focal planes of the sample in relation to the image capturedevice, a sample structure, a convergence value, a pattern, or afrequency determined based at least on one of color analysis, analysisof optical aberrations, computational reconstruction, patternrecognition, Fourier transformation, or light field analysis.

In some embodiments, the sample may be stained and the color analysismay comprise determining the attribute for an area of the sample basedat least on comparing a color or stain structure of the area with acolor or stain structure of another area of the sample. The coloranalysis may comprise determining the attribute for an area of thesample based at least on comparing a color or stain structure of thearea with a color or stain structure from empirical data. The sample maybe stained using any stain, for example, it may be at least one of aRomanowsky stain, a Gram stain, a hematoxylin and eosin (H&E) stain, animmunohistochemistry (IHC) stain, a methylene blue stain, or a DAPIstain.

In some embodiments, the analysis of optical aberrations may compriseidentifying an optical aberration from the initial image set, anddetermining, in response to identifying the optical aberration, theattribute. Determining, in response to identifying the opticalaberration, the attribute may further comprise determining, in responseto identifying the optical aberration, the sample's distance from thelens, and determining the attribute in response to determining thesample's distance from the lens.

In some embodiments, determining the 3D process may comprise determiningthe sample structure based at least on the computational reconstruction,and determining an illumination condition and an image capture settingin response to determining the sample structure. Determining theillumination condition and the image capture setting may compriseidentifying, in response to the computational reconstruction, an area ofthe sample having a convergence value that indicates low convergence.

In some embodiments, the illumination assembly may comprise a laser andidentifying the attribute may comprise identifying the pattern fromilluminating the sample with the laser. Identifying the attributecomprises determining, using pattern recognition, the attribute of thesample.

In some embodiments, identifying the attribute may comprise performingthe Fourier transformation using the initial image set, determiningfrequencies represented in the sample based at least on the Fouriertransformation, and identifying the attribute in response to determiningthe frequencies.

In some embodiments, identifying the attribute may comprise performingthe light field analysis in response to capturing the initial image set,and identifying the attribute by identifying characteristics of thelight field analysis.

In some embodiments, capturing the initial image set may comprisecapturing a first plurality of images, the 3D process may comprisecapturing a second plurality of images, and a number of images in thesecond plurality of images may be greater than a number of images in thefirst plurality of images. The first plurality of images and the secondplurality of images may correspond to a same area of the sample. Thesecond plurality of images may be captured at a same relative locationof the image capture device with respect to the sample as the firstplurality of images. The 3D process may comprise a computational processbased at least on capturing the initial image set.

In some embodiments, the illumination assembly may comprise an LEDarray. The illumination assembly may comprise one or more of a halogenlamp, an LED, an incandescent lamp, a laser or a sodium lamp. Aresolution of the output image set may be higher than a resolution ofthe initial image set.

In some embodiments, the image capture device may comprise a pluralityof image capture devices and the illumination assembly may comprise aplurality of illumination assemblies. The plurality of image capturedevices may comprise a first image capture device and a second imagecapture device and the plurality of illumination assemblies may comprisea first illumination assembly and a second illumination assembly. Thefirst illumination assembly may comprise a light source for backlightillumination and the second illumination assembly may comprise aplurality of light sources. The first image capture device may comprisea preview camera and the second image capture device may comprise amicroscope objective and detector.

In one example, a method for adaptive sampling may comprise anycombination of steps described herein.

Features from any of the above-mentioned embodiments may be used incombination with one another in accordance with the general principlesdescribed herein. These and other embodiments, features, and advantageswill be more fully understood upon reading the following detaileddescription in conjunction with the accompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate a number of exemplary embodimentsand are a part of the specification. Together with the followingdescription, these drawings demonstrate and explain various principlesof the instant disclosure.

FIG. 1 is a diagram of an exemplary microscope, in accordance with someembodiments of the present disclosure.

FIG. 2A is a diagram of the optical paths of two beam pairs when themicroscope of FIG. 1. Is out of focus, in accordance with someembodiments of the present disclosure.

FIG. 2B is a diagram of the optical paths of two beam pairs when themicroscope of FIG. 1 is in focus, in accordance with some embodiments ofthe present disclosure.

FIG. 3 is a workflow diagram showing an exemplary process for adaptivesensing of a sample, in accordance with some embodiments of the presentdisclosure.

FIG. 4 is a flowchart showing an exemplary process for adaptive sensingof a sample, in accordance with some embodiments of the presentdisclosure.

FIG. 5 is a graph corresponding to an initial acquisition of a sample,in accordance with some embodiments of the present disclosure.

FIG. 6A is an illustration of exemplary image data acquired at a firstdepth, in accordance with some embodiments of the present disclosure.

FIG. 6B is an illustration of exemplary image data acquired at a seconddepth, in accordance with some embodiments of the present disclosure.

FIG. 6C is an illustration of exemplary image data acquired at a thirddepth, in accordance with some embodiments of the present disclosure.

FIG. 7A is a diagram of a configuration for a plurality of illuminationconditions, in accordance with some embodiments of the presentdisclosure.

FIG. 7B is a diagram of a configuration for a plurality of illuminationconditions, in accordance with some embodiments of the presentdisclosure.

FIG. 7C is a diagram of a configuration for a plurality of illuminationconditions, in accordance with some embodiments of the presentdisclosure.

FIG. 7D is a diagram of a configuration for a plurality of illuminationconditions, in accordance with some embodiments of the presentdisclosure.

FIG. 7E is a diagram of a configuration for a plurality of illuminationconditions, in accordance with some embodiments of the presentdisclosure.

FIG. 7F is a diagram of a configuration for a plurality of illuminationconditions, in accordance with some embodiments of the presentdisclosure.

FIG. 8A is a top-down view of a sample in accordance with someembodiments of the present disclosure.

FIG. 8B is a side view of the sample of FIG. 8A when the sample is a 2Dsample.

FIG. 8C is a side view of the sample of FIG. 8A when the sample is a2.5D sample.

FIG. 8D is a side view of the sample of FIG. 8A showing a single layerwhen the sample is a 3D sample.

FIG. 8E is a side view of the sample of FIG. 8A showing multiple layerswhen the sample is a 3D sample.

Throughout the drawings, identical reference characters and descriptionsindicate similar, but not necessarily identical, elements. While theexemplary embodiments described herein are susceptible to variousmodifications and alternative forms, specific embodiments have beenshown by way of example in the drawings and will be described in detailherein. However, the exemplary embodiments described herein are notintended to be limited to the particular forms disclosed. Rather, theinstant disclosure covers all modifications, equivalents, andalternatives falling within the scope of the appended claims.

DETAILED DESCRIPTION

The present disclosure is generally directed to systems and methods foradaptive sensing of a sample. As will be explained in greater detailbelow, embodiments of the instant disclosure may be configured toperform image captures using a larger focus depth or at severaldistances from the sample or locations or with added illuminationconditions when it is determined to be necessary for imaging the sample,for example based on a detected attribute of the sample. The acquisitionprocedure and/or the computational process may be adapted. The resultingimages may advantageously comprise most or all of the focus planes withimportant data at every point. Acquisition time may be significantlyreduced by avoiding unnecessary image captures using focal planes whichmay not contribute additional data or by avoiding capturing images withillumination conditions which aren't necessary for the computationalprocess. The user experience may be improved, for example, because thesystem may provide high-quality images without requiring the user todetermine in advance how to adjust the acquisition procedure.

Tomography refers generally to methods where a three-dimensional (3D)sample is sliced computationally into several 2D slices. Confocalmicroscopy refers to methods for blocking out-of-focus light in theimage formation which improves resolution and contrast but tends to leadto focusing on a very thin focal plane and small field of view. Bothtomography and confocal microscopy as well as other methods used in 3Dimaging may be used in conjunction with aspects of the presentdisclosure to produce improved results. Another method may be staggeredline scan sensors, where the sensor has several line scanners atdifferent heights and or angles, and the sensor may take images atseveral focus planes at the same time.

The following will provide, with reference to FIGS. 1-8E, detaileddescriptions of adaptive sensing. FIGS. 1, 2, and 7A-7E illustrate amicroscope and various microscope configurations. FIGS. 3-4 illustrateexemplary processes for adaptive sensing of a sample. FIG. 5 shows anexemplary graph for attribute estimation. FIGS. 6A-6C illustrateexemplary images of a sample at different depths. FIG. 8A-8E illustrateexemplary sample geometries.

FIG. 1 is a diagrammatic representation of a microscope 100 consistentwith the exemplary disclosed embodiments. The term “microscope” as usedherein generally refers to any device or instrument for magnifying anobject which is smaller than easily observable by the naked eye, i.e.,creating an image of an object for a user where the image is larger thanthe object. One type of microscope may be an “optical microscope” thatuses light in combination with an optical system for magnifying anobject. An optical microscope may be a simple microscope having one ormore magnifying lens. Another type of microscope may be a “computationalmicroscope” that comprises an image sensor and image-processingalgorithms to enhance or magnify the object's size or other properties.The computational microscope may be a dedicated device or created byincorporating software and/or hardware with an existing opticalmicroscope to produce high-resolution digital images. As shown in FIG.1, microscope 100 comprises an image capture device 102, a focusactuator 104, a controller 106 connected to memory 108, an illuminationassembly 110, and a user interface 112. An example usage of microscope100 may be capturing images of a sample 114 mounted on a stage 116located within the field-of-view (FOV) of image capture device 102,processing the captured images, and presenting on user interface 112 amagnified image of sample 114.

Image capture device 102 may be used to capture images of sample 114. Inthis specification, the term “image capture device” as used hereingenerally refers to a device that records the optical signals entering alens as an image or a sequence of images. The optical signals may be inthe near-infrared, infrared, visible, and ultraviolet spectrums.Examples of an image capture device comprise a CCD camera, a CMOScamera, a photo sensor array, a video camera, a mobile phone equippedwith a camera, a webcam, a preview camera, a microscope objective anddetector, etc. Some embodiments may comprise only a single image capturedevice 102, while other embodiments may comprise two, three, or evenfour or more image capture devices 102. In some embodiments, imagecapture device 102 may be configured to capture images in a definedfield-of-view (FOV). Also, when microscope 100 comprises several imagecapture devices 102, image capture devices 102 may have overlap areas intheir respective FOVs. Image capture device 102 may have one or moreimage sensors (not shown in FIG. 1) for capturing image data of sample114. In other embodiments, image capture device 102 may be configured tocapture images at an image resolution higher than VGA, higher than 1Megapixel, higher than 2 Megapixels, higher than 5 Megapixels, 10Megapixels, higher than 12 Megapixels, higher than 15 Megapixels, orhigher than 20 Megapixels. In addition, image capture device 102 mayalso be configured to have a pixel size smaller than 15 micrometers,smaller than 10 micrometers, smaller than 5 micrometers, smaller than 3micrometers, or smaller than 1.6 micrometer.

In some embodiments, microscope 100 comprises focus actuator 104. Theterm “focus actuator” as used herein generally refers to any devicecapable of converting input signals into physical motion for adjustingthe relative distance between sample 114 and image capture device 102.Various focus actuators may be used, including, for example, linearmotors, electrostrictive actuators, electrostatic motors, capacitivemotors, voice coil actuators, magnetostrictive actuators, etc. In someembodiments, focus actuator 104 may comprise an analog position feedbacksensor and/or a digital position feedback element. Focus actuator 104 isconfigured to receive instructions from controller 106 in order to makelight beams converge to form a clear and sharply defined image of sample114. In the example illustrated in FIG. 1, focus actuator 104 may beconfigured to adjust the distance by moving image capture device 102.

However, in other embodiments, focus actuator 104 may be configured toadjust the distance by moving stage 116, or by moving both image capturedevice 102 and stage 116. Microscope 100 may also comprise controller106 for controlling the operation of microscope 100 according to thedisclosed embodiments. Controller 106 may comprise various types ofdevices for performing logic operations on one or more inputs of imagedata and other data according to stored or accessible softwareinstructions providing desired functionality. For example, controller106 may comprise a central processing unit (CPU), support circuits,digital signal processors, integrated circuits, cache memory, or anyother types of devices for image processing and analysis such as graphicprocessing units (GPUs). The CPU may comprise any number ofmicrocontrollers or microprocessors configured to process the imageryfrom the image sensors. For example, the CPU may comprise any type ofsingle- or multi-core processor, mobile device microcontroller, etc.Various processors may be used, including, for example, processorsavailable from manufacturers such as Intel®, AMD®, etc. and may comprisevarious architectures (e.g., x86 processor, ARM®, etc.). The supportcircuits may be any number of circuits generally well known in the art,including cache, power supply, clock and input-output circuits.Controller 106 may be at a remote location, such as a computing devicecommunicatively coupled to microscope 100.

In some embodiments, controller 106 may be associated with memory 108used for storing software that, when executed by controller 106,controls the operation of microscope 100. In addition, memory 108 mayalso store electronic data associated with operation of microscope 100such as, for example, captured or generated images of sample 114. In oneinstance, memory 108 may be integrated into the controller 106. Inanother instance, memory 108 may be separated from the controller 106.

Specifically, memory 108 may refer to multiple structures orcomputer-readable storage mediums located at controller 106 or at aremote location, such as a cloud server. Memory 108 may comprise anynumber of random access memories, read only memories, flash memories,disk drives, optical storage, tape storage, removable storage and othertypes of storage.

Microscope 100 may comprise illumination assembly 110. The term“illumination assembly” as used herein generally refers to any device orsystem capable of projecting light to illuminate sample 114.

Illumination assembly 110 may comprise any number of light sources, suchas light emitting diodes (LEDs), LED array, lasers, and lamps configuredto emit light, such as a halogen lamp, an incandescent lamp, or a sodiumlamp. In one embodiment, illumination assembly 110 may comprise only asingle light source. Alternatively, illumination assembly 110 maycomprise four, sixteen, or even more than a hundred light sourcesorganized in an array or a matrix. In some embodiments, illuminationassembly 110 may use one or more light sources located at a surfaceparallel to illuminate sample 114. In other embodiments, illuminationassembly 110 may use one or more light sources located at a surfaceperpendicular or at an angle to sample 114.

In addition, illumination assembly 110 may be configured to illuminatesample 114 in a series of different illumination conditions. In oneexample, illumination assembly 110 may comprise a plurality of lightsources arranged in different illumination angles, such as atwo-dimensional arrangement of light sources. In this case, thedifferent illumination conditions may comprise different illuminationangles. For example, FIG. 1 depicts a beam 118 projected from a firstillumination angle α1, and a beam 120 projected from a secondillumination angle α2. In some embodiments, first illumination angle α1and second illumination angle α2 may have the same value but oppositesign. In other embodiments, first illumination angle α1 may be separatedfrom second illumination angle α2. However, both angles originate frompoints within the acceptance angle of the optics. In another example,illumination assembly 110 may comprise a plurality of light sourcesconfigured to emit light in different wavelengths. In this case, thedifferent illumination conditions may comprise different wavelengths. Inyet another example, illumination assembly 110 may configured to use anumber of light sources at predetermined times. In this case, thedifferent illumination conditions may comprise different illuminationpatterns. Accordingly and consistent with the present disclosure, thedifferent illumination conditions may be selected from a groupincluding: different durations, different intensities, differentpositions, different illumination angles, different illuminationpatterns, different wavelengths, or any combination thereof.

Consistent with disclosed embodiments, microscope 100 may comprise, beconnected with, or in communication with (e.g., over a network orwirelessly, e.g., via Bluetooth) user interface 112. The term “userinterface” as used herein generally refers to any device suitable forpresenting a magnified image of sample 114 or any device suitable forreceiving inputs from one or more users of microscope 100. FIG. 1illustrates two examples of user interface 112. The first example is asmartphone or a tablet wirelessly communicating with controller 106 overa Bluetooth, cellular connection or a Wi-Fi connection, directly orthrough a remote server. The second example is a PC display physicallyconnected to controller 106. In some embodiments, user interface 112 maycomprise user output devices, including, for example, a display, tactiledevice, speaker, etc. In other embodiments, user interface 112 maycomprise user input devices, including, for example, a touchscreen,microphone, keyboard, pointer devices, cameras, knobs, buttons, etc.With such input devices, a user may be able to provide informationinputs or commands to microscope 100 by typing instructions orinformation, providing voice commands, selecting menu options on ascreen using buttons, pointers, or eye-tracking capabilities, or throughany other suitable techniques for communicating information tomicroscope 100. User interface 112 may be connected (physically orwirelessly) with one or more processing devices, such as controller 106,to provide and receive information to or from a user and process thatinformation. In some embodiments, such processing devices may executeinstructions for responding to keyboard entries or menu selections,recognizing and interpreting touches and/or gestures made on atouchscreen, recognizing and tracking eye movements, receiving andinterpreting voice commands, etc.

Microscope 100 may also comprise or be connected to stage 116. Stage 116comprises any horizontal rigid surface where sample 114 may be mountedfor examination. Stage 116 may comprise a mechanical connector forretaining a slide containing sample 114 in a fixed position. Themechanical connector may use one or more of the following: a mount, anattaching member, a holding arm, a clamp, a clip, an adjustable frame, alocking mechanism, a spring or any combination thereof. In someembodiments, stage 116 may comprise a translucent portion or an openingfor allowing light to illuminate sample 114. For example, lighttransmitted from illumination assembly 110 may pass through sample 114and towards image capture device 102. In some embodiments, stage 116and/or sample 114 may be moved using motors or manual controls in the XYplane to enable imaging of multiple areas of the sample.

FIG. 2A and FIG. 2B depict a closer view of microscope 100 in two cases.Specifically, FIG. 2A illustrates the optical paths of two beams pairswhen microscope 100 is out of focus, and FIG. 2B illustrates the opticalpaths of two beams pairs when microscope 100 is in focus. In cases wherethe sample is thicker than the depth of focus or the change in depth israpid, some portions of the sample may be in focus, while other portionsmay not be in focus.

As shown in FIGS. 2A and 2B, image capture device 102 comprises an imagesensor 200 and a lens 202. In microscopy, lens 202 may be referred to asan objective lens of microscope 100. The term “image sensor” as usedherein generally refers to a device capable of detecting and convertingoptical signals into electrical signals. The electrical signals may beused to form an image or a video stream based on the detected signals.Examples of image sensor 200 may comprise semiconductor charge-coupleddevices (CCD), active pixel sensors in complementarymetal-oxide-semiconductor (CMOS), or N-type metal-oxide-semiconductor(NMOS, Live MOS). The term “lens” as used herein refers to a ground ormolded piece of glass, plastic, or other transparent material withopposite surfaces either or both of which are curved, by means of whichlight rays are refracted so that they converge or diverge to form animage. The term “lens” may also refer to an element containing one ormore lenses as defined above, such as in a microscope objective. Thelens is positioned at least generally transversely of the optical axisof image sensor 200. Lens 202 may be used for concentrating light beamsfrom sample 114 and directing them towards image sensor 200. In someembodiments, image capture device 102 may comprise a fixed lens or azoom lens.

When sample 114 is located at a focal-plane 204, the image projectedfrom lens 202 is completely focused. The term “focal-plane” is usedherein to describe a plane that is perpendicular to the optical axis oflens 202 and passes through the lens's focal point. The distance betweenfocal-plane 204 and the center of lens 202 is called the focal lengthand is represented by D1. In some cases, sample 114 may not becompletely flat, and there may be small differences between focal-plane204 and various regions of sample 114. Accordingly, the distance betweenfocal-plane 204 and sample 114 or a region of interest (ROI) of sample114 is marked as D2. The distance D2 corresponds with the degree inwhich an image of sample 114 or an image of ROI of sample 114 is out offocus. For example, distance D2 may be between 0 and about 3 mm. In someembodiments, D2 may be greater than 3 mm. When distance D2 equals tozero, the image of sample 114 (or the image of ROI of sample 114) iscompletely focused. In contrast, when D2 has a value other than zero,the image of sample 114 (or the image of ROI of sample 114) is out offocus.

FIG. 2A depicts a case where the image of sample 114 is out of focus.For example, the image of sample 114 may be out of focus when the beamsof light received from sample 114 do not converge on image sensor 200.FIG. 2A depicts a beams pair 206 and a beams pair 208. Neither pairconverges on image sensor 200. For the sake of simplicity, the opticalpaths below sample 114 are not shown. Consistent with the presentdisclosure, beams pair 206 may correspond with beam 120 projected fromillumination assembly 110 at illumination angle α2, and beams pair 208may correspond with beam 118 projected from illumination assembly 110 atillumination angle α1. In addition, beams pair 206 may concurrently hitimage sensor 200 with beams pair 208. The term “concurrently” in thiscontext means that image sensor 200 has recorded information associatedwith two or more beams pairs during coincident or overlapping timeperiods, either where one begins and ends during the duration of theother, or where a later one starts before the completion of the other.In other embodiments, beams pair 206 and beams pair 208 may sequentiallycontact image sensor 200. The term “sequentially” means that imagesensor 200 has started recording information associated with, forexample, beam pair 206 after the completion of recording informationassociated with, for example, beam pair 208.

As discussed above, D2 is the distance between focal-plane 204 andsample 114, and it corresponds with the degree in which sample 114 isout of focus. In one example, D2 may have a value of 50 micrometers.Focus actuator 104 is configured to change distance D2 by convertinginput signals from controller 106 into physical motion. In someembodiments, in order to focus the image of sample 114, focus actuator104 may move image capture device 102. In this example, to focus theimage of sample 114 focus actuator 104 may move image capture device 10250 micrometers up. In other embodiments, in order to focus the image ofsample 114, focus actuator 104 may move stage 116 down. Therefore, inthis example, instead of moving image capture device 102 50 micrometersup, focus actuator 104 may move stage 116 50 micrometers down.

FIG. 2B illustrates a case where the image of sample 114 is in focus. Inthis case, both beam pairs 206 and 208 converge on image sensor 200, anddistance D2 equals to zero. In other words, focusing the image of sample114 (or the image of ROI of sample 114) may rely on adjusting therelative distance between image capture device 102 and sample 114. Therelative distance may be represented by D1-D2, and when distance D2equals to zero, the relative distance between image capture device 102and sample 114 equals to distance D1, which means that the image ofsample 114 is focused.

FIG. 3 illustrates an exemplary workflow 300 for adaptive sensing of asample, such as sample 114, using a microscope, such as microscope 100.Samples on a slide may have varying or otherwise non-homogeneousthickness and/or flatness, or a homogenous thickness but unknown priorto the acquisition, which may complicate imaging the samples. Samplesmay have varying thicknesses based on type of sample. Examples ofsamples include but are not limited to blood, fine needle aspirates(FNA), histopathology tissue samples, frozen sections, sperm, PAP smear,fecal, urine, petroleum, cream, algae, and brain slices. Blood samplesmay be smeared, which may result in some areas being thinner or thickerthan other areas of the sample. Thin samples may be approximately 2microns in thickness. Thick areas may be approximately 10 or moremicrons.

FNA samples may be made using fluid and/or soft material from a syringe,which may be sprayed and smeared on a slide. FNA samples may typicallyhave randomly dispersed thick areas, which may be more difficult toimage and analyze. FNA samples may have thicknesses ranging from 2microns to 20 or more microns.

Tissue samples used for histopathology may typically be embedded withparaffin. These tissue samples may be cut with a microtome. These tissuesamples may have thicknesses ranging from two microns to ten microns,with three microns to six microns being very common. However, they mayoccasionally be cut thicker.

Frozen section tissue samples may be frozen in order to facilitateprocessing. Frozen section tissue samples may be thicker thanhistopathology samples due to technical difficulties. Althoughthicknesses ranging from four microns to six microns may be desirable,samples may be ten microns to thirty or more microns.

Brain slices may sometimes be cut thicker than other sample types. Forexample, brain slices may have thicknesses of 5-50 microns, but may evenhave thicknesses ranging from 50-100 microns, and may be as thick as 200microns.

FIGS. 8A-8E illustrate possible sample geometries of a sample 814, whichmay correspond to sample 114. Sample 814 may be mounted on a slide 815,which may be a slide suitable for mounting on a microscope. FIG. 8Ashows a top-down view of sample 814, which also illustrates how thesample geometry may not be easily determined without an initialacquisition of the sample. In some embodiments, the microscope comprisesa depth of field inside the field of view of the microscope, and thesample is imaged along a layer 817 corresponding to the depth of fieldof the microscope. The entire area of the sample imaged at a givenmoment can be within layer 817, e.g. a 2D sample, or the sample canextend beyond the layer 817 corresponding to the depth of field of themicroscope, e.g. 2.5 D and 3D samples.

FIG. 8B shows sample 814 having a 2D geometry. In this example, a singlelayer 817 may be used to capture sample 814. Layer 817 may correspond toa depth of field of a focal plane. FIG. 8C shows sample 814 having a2.5D geometry. As seen in FIG. 8C, a single layer 817 may capture muchof sample 814 in focus, but may miss some portions, such as the peaksand valleys extending outside of layer 817. To properly capture sample814 in FIG. 8C, a 2.5D reconstruction may rely on additional imagescorresponding to the portions outside of layer 817. For instance,additional layers 817 may be used. However, additional efficiency may begained by determining where the peaks and valleys are located (e.g.,based on an attribute of sample 814) to specifically target the missingportions, according to embodiments described herein. FIGS. 8D and 8Eshow sample 814 having a 3D geometry. In FIG. 8D, a single layer 817 maybe sufficient to capture the information in sample 814 which is ofinterest for a certain case. The 3D process may be used to enableimaging that layer without interference from the other parts of thesample. As seen in FIG. 8E, multiple layers 817 may be used to cover thedepth range of sample 814. A 3D reconstruction may reconstruct layers817 to produce an output image for sample 814. However, additionalefficiency may be gained by specifically selecting layers 814 based onthe attribute of sample 814. For example, the topmost layer 817 may onlycover a small portion of sample 814, and therefore may be reduced insize.

At block 310, the microscope performs an initial acquisition of thesample. The microscope may capture images of the sample at a singledepth 320A, multiple depths 320B, and/or multiple locations 320C. Singledepth 320A may comprise capturing images using one or more illuminationsat a single depth of the sample. Similarly, multiple depths 320B maycomprise capturing images using one or more illuminations at multipledepths of the sample. Multiple locations 320C may comprise capturingimages using one or more illuminations at multiple locations of thesample.

The sample may be scanned with some granularity in order to estimate anattribute, such as thickness, depth, and/or density, at differentlocations and to further determine a benefit from acquiring and/orcalculating information about a larger depth of the sample. In addition,the acquired images may be part of a computational reconstruction of thesample, using a computational reconstruction algorithm such a twodimensional (2D), 2.5D, or 3D reconstruction to analyze the sample.Whereas 3D reconstruction may reconstruct a sample having 3D shapesand/or features, 2.5D reconstruction may reconstruct a sample having agenerally thin, flat shape but uneven or non-smooth, similar to acrumpled sheet of paper. An example of that may be a histopathologyslice of a tissue, which may not have been placed completely flat on thesurface of the slide.

At block 330, the microscope may calculate an attribute of the sample,which could indicate that the sample has 2.5D or 3D structure, andtherefore may require more than one focal plane. The attribute may becalculated for a single location 340A, multiple locations in a field ofview (“FOV”) 340B, and/or multiple FOVs 340C. The attribute may comprisea thickness, a depth, a density, a color, a stain structure of thesample (e.g., a structure formed or made visible when sample isstained), one or more distances from a lens, one or more focal planes ofthe sample, a sample structure of the sample, a convergence value, apattern, or a frequency represented by the sample, in one or morelocations of the sample. The attribute may be calculated in variousways.

The depth of the sample may be determined from physical attributes ofthe acquisition or the image attributes resulting from the physicalattributes of, for instance, defocus. For example, a sharpness level inthe image at one location may be compared with a sharpness level atother locations of the captured image or a threshold sharpness value. Inanother example, the sample may have been illuminated from differentangles. A difference in shift of features under the differentillumination angles may be used to calculate the depth.

The depth of the sample, or the benefit from additional depthinformation may be determined from the contents of the image. Forexample, a particular feature or object, such as a specific type of cell(e.g., white blood cells in a blood smear sample) may be recognized inan area of the image. The recognized feature may indicate the depth,and/or may indicate the benefit from additional depth information. Forexample, the feature may be larger than what was initially captured ormay be a feature of interest. The acquisition process may be adapted toacquire additional images in several focus planes near the feature oracquire information to perform a 3D reconstruction in order to assurethe feature is in perfect focus and to obtain the 3D information at thatlocation.

The sample may be stained using, for example, a Romanowsky stain, a Gramstain, a hematoxylin and eosin (H&E) stain, an immunohistochemistry(IHC) stain, a methylene blue stain, a DAPI stain, a fluorescent stain,or any other suitable stain.

Image analysis may be used, for example on preview or magnified images,to determine the attribute. A color, a contrast, an edge, a diffraction,and/or an absorption may be analyzed in the images. Different parts ofthe sample may respond differently to staining or illumination withdifferent wavelengths or may otherwise appear different in imaging dueto changes in focus and/or other characteristics. These differences maybe used to estimate which areas of the sample are thicker or thinnerthan others. For example, if an area is darker than its surroundings,the area may be thicker than its surroundings. Moreover, knowledge frompreviously analyzed samples and/or prior analysis may be available. Forexample, some samples made by a consistent process, such as bloodsamples made with a stainer. More accurate results may be achieved byusing the prior analysis.

The attribute may be determined from changes in aberrations. Whensamples are in different focal planes, the optics may introducedifferent optical aberrations. These aberrations may produce artifactsin the images, such as stretching, contracting, shifting, etc. Theaberrations may suggest the sample's distance from the lens, which maybe used to estimate the depth and/or thickness of the sample. Inaddition, the pupil function of the optics at each location may be useddirectly.

The attribute may be determined using computational reconstruction. Forexample, the sample may be analyzed using the results from acomputational reconstruction algorithm (such as 2D, 2.5D, or 3Dreconstruction) applied to the initial acquisition. For instance, 3Dreconstruction may be used with a fewer number of layers than would beutilized for a complete 3D reconstruction in order to assess a structureof the sample. Additional layers and/or illumination conditions as wellas locations for the additional layers may be determined based on thestructure of the sample.

The attribute may be determined using values of convergence.Reconstruction algorithms may track the algorithm's convergence. Placesof low convergence may indicate thick or complex areas which may benefitfrom further imaging.

The attribute may be determined using a laser. The sample may beilluminated using one or more lasers. The resulting patterns may beanalyzed to determine the attribute.

The attribute may be determined using pattern recognition, such ascomputer vision or deep learning. For known types of samples, a databaseof expected patterns and shapes in images may be available. Forinstance, a white blood cell's appearance when covered by differentthicknesses of other fluids in an FNA sample may be available and asubsequent FNA sample may be analyzed based on the expected patterns todetermine the attribute.

The attribute may be determined from the properties of a Fouriertransformation of the images. The Fourier transformation of the imagesmay reveal the frequencies represented in the images. Thick samples mayhave less high frequencies because of the multiple scatters of light. An“envelope” of decreased of energy with increasing frequencies maysuggest thickness. A change of frequencies between adjacent tiles of animage may indicate differences between the tiles.

The attribute may be determined from digital refocusing using lightfield analysis. Light field analysis may be based on knowledge of theconditions under which the images were taken to form a basic refocusingapproximation. The resultant refocusing approximation may be used toestimate the thickness and location of the sample.

The attribute may be determined from the statistics of lateral shifts inareas of the images. Changes in focus may result in lateral shifts ofdetails. Analyzing how changes in focus affect different areas in theimages may be used to determine local depth.

The attribute may be determined using machine learning (e.g., neuralnetworks, convolutional neural networks, deep learning, computer vision,etc.). A database of images under known acquisition conditions may beused to train a machine learning algorithm to identify the thicknessand/or location of the sample.

At block 350, the microscope may determine a process for adapting theacquisition process. The process may be an acquisition procedure thatmay perform 3D measurement, such as focus stacking or 3D reconstruction.The process may comprise a 3D reconstruction on initial acquisition360A, which in some embodiments may be a continuation of the initialacquisition. In other words, the initial acquisition may be part of the3D reconstruction process. The process may comprise a 3D acquisition360B, such as focus stacking. The process may be performed after orconcurrently with the attribute estimation.

The attribute, such as thickness and/or density in different locationsof the sample, may be used to determine where to place additional layersand/or voxels for 2.5D analysis or 3D analysis. 3D acquisition maycomprise, for example, focus stack on a same lateral area 370A, focusstack in layers 370B, and/or a single distance from the sample 370C.Thus, the 3D acquisition 360B may be able to analyze different focalplanes in a field of view, expand autofocus parameters for 2.5D and 3Dsamples, and determine a location of maximum data in a thick sample etc.

At block 380, 3D data may be prepared. For example, the 3D data may beregistered so the layers are in the correct relative locations or useimage processing to improve the display or creating an “all in focus”image for cases such as 2.5D samples. When the 3D data has beenprepared, the data may be displayed or stored as shown at block 390A.The 3D computational reconstruction can be performed from any subset ofimages taken as shown at block 390B. For example, a 3D reconstructionprocess may be started, modified, and completed.

The workflow 300 may comprise a method for adaptive sensing of thesample, and each of the blocks of workflow 300 may comprise steps of themethod for adaptive sensing of the sample.

FIG. 4 is a flowchart of an exemplary process 400 for determiningadaptive sensing, for example, sample 114. The steps of process 400 maybe performed by a microscope, such as an autofocus microscope. The term“autofocus microscope” as used herein generally refers to any device formagnifying sample 114 with the capability to focus the image of sample114 (or the image of ROI of sample 114) in an automatic or semiautomaticmanner. In the following description, reference is made to components ofmicroscope 100 for purposes of illustration. It will be appreciated,however, that other implementations are possible and that othercomponents may be utilized to implement the example process.

At step 410, image capture device 102 may be used to capture an initialimage set of sample 114. Capturing the initial image set may comprisecapturing more than one image, such as a first plurality of images. Forexample, a plurality of illumination conditions for illuminationassembly 110 to illuminate sample 114. Examples or illuminationconditions comprise illumination angles, illumination wavelengths, orillumination patterns. The illumination conditions and image capturesettings may be default settings for microscope 100, or may be defaultsettings based on the type of sample 114. The default settings maycorrespond to settings which may minimize how many initial images arecaptured to determine an attribute of sample 114. The default settingsmay correspond to preliminary steps of a 3D measurement process.Alternatively, the default settings may be determined by a user.

At step 420, controller 106 may identify, in response to the initialimage set, an attribute of sample 114. The attribute may correspond toone or more aspects of sample 114. In some cases, several attributes maybe determined, for example, the depth (e.g., distance of the sample orcertain layers of the sample from the microscope) and thickness of thesample in the analysis process may be determined. The attribute maycomprise a thickness of sample 114 at one or more locations. Theattribute may comprise a depth of sample 114 at one or more locations.The attribute may comprise a density of sample 114 at one or morelocations. Other attributes of sample 114 may comprise a color, a stainstructure of sample 114 (e.g., a structure formed or made visible whensample 114 is stained), a distance from lens 202 of image capture device102, a plurality of distances between lens 202 and sample 114, aplurality of focal planes of sample 114 in relation to image capturedevice 102, a sample structure of sample 114, a convergence value, apattern, or a frequency determined based at least on one of coloranalysis, analysis of optical aberrations, computational reconstruction,pattern recognition, Fourier transformation, or light field analysis.

In some embodiments, sample 114 may be stained using, for example, aRomanowsky stain, a Gram stain, a hematoxylin and eosin (H&E) stain, animmunohistochemistry (IHC) stain, a methylene blue stain, a DAPI stain,a fluorescent stain, or any other suitable stain. When sample 114 isstained, controller 106 may perform color analysis to determine theattribute of sample 114. For example, the attribute for an area ofsample 114 may be determined based on comparing a color or stainstructure of the area with a color or stain structure of another area ofthe sample. In other examples, the attribute for the area may bedetermined based on comparing the color or stain structure of the areawith a color or stain structure from empirical data. For instance,memory 108 may comprise a database of colors and/or stain structures andhow they may correlate to attributes. The comparison of colors and/orstain structures may be based on pattern recognition, machine learning,or any other suitable comparison process.

In some embodiments, controller 106 may perform analysis of opticalaberrations to determine the attribute of sample 114. This analysis maycomprise identifying an optical aberration from the initial set anddetermining, in response to identifying the optical aberration, theattribute. For example, controller 106 may determine a distance ofsample 114 from lens 202 in response to identifying the opticalaberration. Controller 106 may then determine the attribute in responseto determining this distance.

In some embodiments, identifying the attribute may comprise usingpattern recognition. For example, illumination assembly 110 mayilluminate sample 114 with a laser, and controller 106 may performpattern recognition on the resulting pattern to identify the attribute.

In some embodiments, identifying the attribute may comprise performing aFourier transformation. For example, controller 106 may perform theFourier transformation using the initial image set. Controller 106 maythen determine frequencies represented in sample 114 based at least onthe Fourier transformation. Controller 106 may identify the attribute inresponse to determining the frequencies.

In some embodiments, identifying the attribute may comprise performing alight field analysis. For example, in response to capturing the initialimage set, controller 106 may perform the light field analysis. Theinitial image set may have been illuminated under particularillumination settings, for example by a laser of illumination assembly110. Controller 106 may then identify the attribute by identifyingcharacteristics of the light field analysis.

In addition, calculating the attribute may comprise estimating theattribute corresponding to one or more locations of sample 114 asdescribed herein. For example, the attribute may refer to one or morespecific points and/or areas of sample 114.

At step 430, controller 106 may determine, in response to identifyingthe attribute, a three-dimensional (3D) process for sensing sample 114.The 3D process may comprise a process for sensing the sample. In someembodiments, the 3D process may comprise a reconstruction process, suchas a 3D reconstruction process, for reconstructing the sample inresponse to the initial image set. In some examples, the 3D process maynot require images beyond the initial image set.

In some embodiments, determining the 3D process may comprise controller106 determining the sample structure based at least on the computationalreconstruction, and determining an illumination condition and an imagecapture setting in response to determining the sample structure.

The term “sample structure” as used herein may refer to a structureand/or structural features of the sample, including, for example,biomolecules, whole cells, portions of cells such as various cellcomponents (e.g., cytoplasm, mitochondria, nucleus, chromosomes,nucleoli, nuclear membrane, cell membrane, Golgi apparatus, lysosomes),cell-secreted components (e.g., proteins secreted to intercellularspace, proteins secreted to body fluids, such as serum, cerebrospinalfluid, urine), microorganisms, and more. Controller 106 may determinethe illumination condition and the image capture setting based on abenefit from capturing additional detail at a location of the samplestructure. Determining the illumination condition and the image capturesetting may also comprise identifying, in response to the computationalreconstruction, an area of the sample having a convergence value thatindicates low convergence.

At step 440, controller 106 may generate, in response to identifying theattribute, an output image set comprising more than one focal planeusing a three-dimensional (3D) process. The 3D process may comprise acomputational process based at least on capturing the initial image set.For example, the 3D process may be a continuation of a computationalprocess that started with capturing the initial image set.

In some embodiments, the 3D process may comprise capturing one or moresubsequent images of the sample using one or more illuminationconditions of illumination assembly 110 and one or more image capturesettings for image capture device 102. A number of illuminationconditions for the 3D process may be greater than a number ofillumination conditions for capturing the initial image set.

In some embodiments, the 3D process may comprise determining a pluralityof focal planes for capturing the one or more subsequent images based atleast on the attribute. The one or more illumination conditions and theone or more image capture settings may correspond to the plurality offocal planes. The one or more subsequent images may be taken at one ormore locations of the sample determined based at least on the attribute.For example, the attribute may indicate one or more ROIs. Controller 106may have determined focal planes to ensure that all ROIs can be capturedin focus.

In some embodiments, the 3D process may comprise performing a 3Dreconstruction of the sample based at least on a subset of imagescaptured by the image capture device. In some embodiments, the 3Dprocess may comprise performing a 2.5D reconstruction of the samplebased at least on a subset of images captured by the image capturedevice in order to generate 3D data from the sample. In other words, the3D process may not be restricted to 3D reconstructions.

In some embodiments, the 3D process comprises performing focus stackingfor the sample based at least on a subset of images captured by theimage capture device. For example, the 3D process may comprise capturinga plurality of images at a respective plurality of focal planes. In someembodiments, the 3D process may comprise capturing images at a pluralityof distances between the image capture device and the sample. A numberof focal planes for the 3D process may be greater than a number ofdistances in the plurality of distances. However, controller 106 maydetermine the number of focal planes to efficiently capture ROIs ofsample 114, without using extraneous focal planes and also withoutneglecting any ROI. In some embodiments, the 3D process may comprise aplurality of focus levels for adjusting the image capture device.

In some embodiments, the 3D process may comprise capturing one or moresubsequent images of the sample using a plurality of illuminationconditions for illuminating the sample. The plurality of illuminationconditions may comprise at least one of an illumination angle, anillumination wavelength, or an illumination pattern. The illuminationconditions may be determined based on the attribute, for example, toensure that ROIs are properly captured with sufficient detail.

In some embodiments, the 3D process may comprise capturing a secondplurality of images. A number of images in the second plurality ofimages may be greater than a number of images in the first plurality ofimages. In some embodiments, the first plurality of images and thesecond plurality of images may correspond to a same area of the sample.In some embodiments, the second plurality of images may be captured at asame relative location of the image capture device with respect to thesample as the first plurality of images. For example, controller 106 maycapture a minimal number of images for the initial image set in order tominimize extraneous captures. Controller 106 may detect the attributeand determine that the initial image set insufficiently captures theROIs of sample 114. Controller 106 may then determine how manyadditional images can be used to sufficiently capture sample 114.

In some embodiments, the 3D process may be performed within a thresholdtime from capturing the initial image set. The threshold time may be oneof 5 microseconds, 10 microseconds, 1 second, 5 seconds, 10 seconds, 1minute, 3 minutes or 5 minutes, or within a range defined by any two ofthe preceding values. For example, the threshold time can be within arange from 5 microseconds to 10 second, from 5 microseconds to 1 minute,from 5 microseconds to 3 minutes, or 5 microseconds to 5 minutes. Insome embodiments, the 3D process may be performed concurrently withcapturing the initial image set, or the 3D process may comprise theinitial image set.

Any of the steps of method 400 can be combined with any method stepcorresponding to a block of workflow 300 as described herein.

FIG. 5 illustrates a graph 500 which may represent an identifiedattribute. The initial image set may comprise images taken at variousheights, corresponding to the height axis or may comprise a set ofimages corresponding to illumination conditions. The data density axismay correspond to the attribute. For example, higher values along thedata density axis may indicate greater complexity or detail present atthat height, which may benefit from additional imaging to properlycapture sufficient detail or merit adding a layer in a focus stack or 3Dprocess. The density may indicate the depth of layers in the sample andthe span of depths may show the thickness of the sample. The processormay be configured to image regions of the sample with higher datadensity at higher resolution. Alternatively or in combination, theprocessor can be configured to process data from the regions of thesample with increased data density in order to provide images withsufficient resolution. Also, the process can be configured not to imageregions of the image with lower data density, for example regions of theimage below a threshold amount. This can provide images with increasedresolution, proper representation of the layers of data, and decreasedscan time and storage for the images.

The data density may correspond to sample structure. The image capturesettings may be based on, for example, heights corresponding to localpeaks in the graph. By prioritizing image capture based on the localpeaks, detailed areas of sample 114 may be captured. In other words, thelocal peaks in FIG. 5 may correspond to ROIs, and the 3D process can bedirected toward the peaks of data density in the attribute.

FIGS. 6A to 6C show images from a 3D reconstruction based on theattribute shown in graph 500. Each image corresponds to a layer in thesample. As can be seen in the figures, there may be good agreementbetween the data density in the graph and the data in the images of thesample. These images may also be used to demonstrate how the attributemay be determined from a focus stack. In this example, the images may beimages taken at different heights, as part of the initial image set.Capturing images at different focal planes may capture differentfeatures, which may be used to determine the attribute of the sample.For example, in FIG. 6A, a feature on the left of the image may be infocus and sufficiently captured, whereas a feature on the upper right ofthe image may be out of focus, insufficiently captured, or tissue fromthe sample not present at the corresponding depth. In contrast, in FIG.6C, the feature on the upper right may be in focus and sufficientlycaptured, whereas the feature on the left may be out of focus,insufficiently captured, or tissue from the sample not present at thecorresponding depth. The images shown in FIG. 6B is at an intermediatedepth as compared with the depth of images FIGS. 6A and 6C. Thus, bydetermining the attribute comprising the data density in the initialimage set at different locations, the 3D process can be adapted to imageregions with higher amounts of data density in order to provide anoutput image with higher resolution and decreased time. Although FIGS.6A to 6C are shown with respect to a substantially fixed laterallocation, additional lateral locations can be scanned with a mosaic ortile pattern or a line scan or staggered line scan at each of aplurality of depths in order to determine the attribute and regions tobe subsequently scanned with the adaptive 3D process as describedherein.

FIGS. 7A-7F illustrate different configurations of microscope 100 fordetermining phase information under a variety of illuminationconditions. According to one embodiment that may be implemented in theconfiguration of FIG. 7A, microscope 100 may comprise illuminationassembly 110, focus actuator 104, lens 202, and image sensor 200. Inthis embodiment, controller 106 may acquire a group of images fromdifferent focal-planes for each illumination condition, which may becompared to determine the attribute.

According to another embodiment that may be implemented in theconfiguration of FIG. 7B, microscope 100 may comprise illuminationassembly 110, lens 202, a beam splitter 600, a first image sensor 200A,and a second image sensor 200B. In this embodiment, first image sensor200A and second image sensor 200B may capture different types of images,and controller 106 may combine the information from first image sensor200A and second image sensor 200B. In one example, image sensor 200A maycapture Fourier-plane images and second image sensor 200B may capturereal-plane images. Accordingly, controller 106 may acquire, for eachillumination condition, a Fourier-plane image from first image sensor200A and a real-plane image from second image sensor 200B. Therefore,controller 106 may combine information from the Fourier-plane image andthe real-plane image in order to determine the attribute. In anotherexample, image sensor 200A may be configured to capture focused imagesand second image sensor 200B is configured to capture unfocused images.It is also possible that additional sensors may be added. For example,2, 3 or more different sensors may be configured to capture images in 2,3 or more different focal planes simultaneously.

According to another embodiment that may be implemented in theconfigurations of FIG. 7C and FIG. 7D, microscope 100 may comprise alight source 602, a beam splitter 600, lens 202, and image sensor 200.In this embodiment, light source 602 may project a light beam (coherentor at least partially coherent) towards beam splitter 600, the beamsplitter generates two light beams that travel through two differentoptical paths and create an interference pattern. In the configurationof FIG. 7C, the interference pattern is created on sample 114, and inFIG. 7D, the interference pattern is created on image sensor 200. In thecase presented in FIG. 7D, controller 106 may identify, for eachillumination condition, the interference pattern between the two lightbeams traveling through the different optical paths, and determine, fromthe interference pattern, the attribute.

According to yet another embodiment that may be implemented in theconfigurations of FIG. 7E and FIG. 7F, microscope 100 may compriseillumination assembly 110, lens 202, an optical element 604, and atleast one image sensor 200. In this embodiment, optical element 604 isconfigured to impose some form of modulation on the light received fromsample 114. The modulation may be imposed on the phase, the frequency,the amplitude, or the polarization of the beam. In the configurationillustrated in FIG. 7E, microscope 100 may comprise a dynamic opticalelement, such as spatial light modulator (SLM), that may dynamicallychange the modulation. Controller 106 may use the different informationcaused by the dynamic optical element to determine the attribute.

Alternatively, in the configuration illustrated in FIG. 7F, microscope100 may comprise a fixed optical element, such as phase-shift mask, beamsplitter 600, first image sensor 200A, and second image sensor 200B.Controller 106 may combine information from first image sensor 200A andsecond image sensor 200B to determine the phase information under eachillumination condition.

As detailed above, the computing devices and systems described and/orillustrated herein broadly represent any type or form of computingdevice or system capable of executing computer-readable instructions,such as those contained within the modules described herein. In theirmost basic configuration, these computing device(s) may each comprise atleast one memory device and at least one physical processor.

The term “memory” or “memory device,” as used herein, generallyrepresents any type or form of volatile or non-volatile storage deviceor medium capable of storing data and/or computer-readable instructions.In one example, a memory device may store, load, and/or maintain one ormore of the modules described herein. Examples of memory devicescomprise, without limitation, Random Access Memory (RAM), Read OnlyMemory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives(SSDs), optical disk drives, caches, variations or combinations of oneor more of the same, or any other suitable storage memory.

In addition, the term “processor” or “physical processor,” as usedherein, generally refers to any type or form of hardware-implementedprocessing unit capable of interpreting and/or executingcomputer-readable instructions. In one example, a physical processor mayaccess and/or modify one or more modules stored in the above-describedmemory device. Examples of physical processors comprise, withoutlimitation, microprocessors, microcontrollers, Central Processing Units(CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcoreprocessors, Application-Specific Integrated Circuits (ASICs), portionsof one or more of the same, variations or combinations of one or more ofthe same, or any other suitable physical processor.

Although illustrated as separate elements, the method steps describedand/or illustrated herein may represent portions of a singleapplication. In addition, in some embodiments one or more of these stepsmay represent or correspond to one or more software applications orprograms that, when executed by a computing device, may cause thecomputing device to perform one or more tasks, such as the method step.

In addition, one or more of the devices described herein may transformdata, physical devices, and/or representations of physical devices fromone form to another. For example, one or more of the devices recitedherein may receive image data of a sample to be transformed, transformthe image data, output a result of the transformation to determine a 3Dprocess, use the result of the transformation to perform the 3D process,and store the result of the transformation to produce an output image ofthe sample. Additionally or alternatively, one or more of the modulesrecited herein may transform a processor, volatile memory, non-volatilememory, and/or any other portion of a physical computing device from oneform to another by executing on the computing device, storing data onthe computing device, and/or otherwise interacting with the computingdevice.

The term “computer-readable medium,” as used herein, generally refers toany form of device, carrier, or medium capable of storing or carryingcomputer-readable instructions. Examples of computer-readable mediacomprise, without limitation, transmission-type media, such as carrierwaves, and non-transitory-type media, such as magnetic-storage media(e.g., hard disk drives, tape drives, and floppy disks), optical-storagemedia (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), andBLU-RAY disks), electronic-storage media (e.g., solid-state drives andflash media), and other distribution systems.

The process parameters and sequence of the steps described and/orillustrated herein are given by way of example only and can be varied asdesired. For example, while the steps illustrated and/or describedherein may be shown or discussed in a particular order, these steps donot necessarily need to be performed in the order illustrated ordiscussed. The various exemplary methods described and/or illustratedherein may also omit one or more of the steps described or illustratedherein or comprise additional steps in addition to those disclosed.

The processor as disclosed herein can be configured to perform any oneor more steps of a method as disclosed herein.

The preceding description has been provided to enable others skilled inthe art to best utilize various aspects of the exemplary embodimentsdisclosed herein. This exemplary description is not intended to beexhaustive or to be limited to any precise form disclosed. Manymodifications and variations are possible without departing from thespirit and scope of the instant disclosure. The embodiments disclosedherein should be considered in all respects illustrative and notrestrictive. Reference should be made to the appended claims and theirequivalents in determining the scope of the instant disclosure.

Unless otherwise noted, the terms “connected to” and “coupled to” (andtheir derivatives), as used in the specification and claims, are to beconstrued as permitting both direct and indirect (i.e., via otherelements or components) connection. In addition, the terms “a” or “an,”as used in the specification and claims, are to be construed as meaning“at least one of” Finally, for ease of use, the terms “including” and“having” (and their derivatives), as used in the specification andclaims, are interchangeable with and have the same meaning as the word“comprising.” The terms “based on” and “in response to” are usedinterchangeably in the present disclosure.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. A microscope comprising: an illuminationassembly; an image capture device configured to collect light from asample illuminated by the illumination assembly; and a processorconfigured to execute instructions which cause the microscope to:capture, using the image capture device, an initial image set of thesample; identify, by the processor in response to the initial image set,an attribute of the sample by performing analysis on the initial imageset independently from a user, wherein the attribute is indicative ofthe sample extending beyond a single focal plane; determine, in responseto identifying the attribute, a three-dimensional (3D) process bydetermining, using the attribute, that the sample extends beyond thesingle focal plane and selecting the 3D process, from a plurality of 3Dprocesses, capable of capturing the sample with more than one focalplane; and generate, using the determined 3D process, an output imageset comprising the more than one focal plane.
 2. The microscope of claim1, wherein the 3D process comprises capturing one or more subsequentimages of the sample using one or more illumination conditions of theillumination assembly and one or more image capture settings for theimage capture device.
 3. The microscope of claim 2, wherein a number ofillumination conditions for the 3D process is greater than a number ofillumination conditions for capturing the initial image set.
 4. Themicroscope of claim 2, wherein the 3D process comprises determining aplurality of focal planes for capturing the one or more subsequentimages based at least on the attribute, and the one or more illuminationconditions and the one or more image capture settings correspond to theplurality of focal planes.
 5. The microscope of claim 2, wherein the oneor more subsequent images are taken at one or more locations of thesample determined based at least on the attribute.
 6. The microscope ofclaim 2, wherein the 3D process comprises performing a 3D reconstructionof the sample based at least on a subset of images captured by the imagecapture device.
 7. The microscope of claim 2, wherein the 3D processcomprises performing a 2.5D reconstruction of the sample based at leaston a subset of images captured by the image capture device in order togenerate 3D data from the sample.
 8. The microscope of claim 2, whereinthe 3D process comprises performing focus stacking for the sample basedat least on a subset of images captured by the image capture device. 9.The microscope of claim 8, wherein the 3D process comprises capturing aplurality of images at a respective plurality of focal planes.
 10. Themicroscope of claim 1, wherein identifying the attribute comprisesestimating the attribute corresponding to one or more locations of thesample.
 11. The microscope of claim 1, wherein the 3D process isperformed within a threshold time from capturing the initial image set.12. The microscope of claim 1, wherein the attribute comprises athickness of the sample at one or more locations of the sample.
 13. Themicroscope of claim 1, wherein the attribute comprises a depth of thesample at one or more locations of the sample.
 14. The microscope ofclaim 1, wherein the 3D process comprises capturing images at aplurality of distances between the image capture device and the sample,and a number of focal planes for the 3D process is greater than a numberof distances in the plurality of distances.
 15. The microscope of claim1, wherein capturing the initial image set comprises capturing theinitial image set of the sample using a plurality of illuminationconditions for illuminating the sample, and the plurality ofillumination conditions comprises at least one of an illumination angle,an illumination wavelength, or an illumination pattern.
 16. Themicroscope of claim 1, wherein the 3D process comprises capturing one ormore subsequent images of the sample using a plurality of illuminationconditions for illuminating the sample, and the plurality ofillumination conditions comprises at least one of an illumination angle,an illumination wavelength, or an illumination pattern.
 17. Themicroscope of claim 1, wherein the 3D process comprises a plurality offocus levels for adjusting the image capture device.
 18. The microscopeof claim 1, wherein the attribute comprises at least one of a thickness,a density, a depth, a color, a stain structure of the sample, a distancefrom a lens of the image capture device, a plurality of distancesbetween the lens of the image capture device and the sample, a pluralityof focal planes of the sample in relation to the image capture device, asample structure, a convergence value, a pattern, or a frequencydetermined based at least on one of color analysis, analysis of opticalaberrations, computational reconstruction, pattern recognition, Fouriertransformation, or light field analysis.
 19. The microscope of claim 18,wherein the sample is stained and the color analysis comprisesdetermining the attribute for an area of the sample based at least oncomparing a color or stain structure of the area with a color or stainstructure of another area of the sample.
 20. The microscope of claim 19,wherein identifying the attribute comprises determining, using patternrecognition, the attribute of the sample.